Most people approaching retirement know that the later you start taking benefits, the more you'll get.

But delay isn't always so easy. While experts blithely suggest working longer, many people with physically demanding jobs can't keep at it until 66 or 70, and others have the decision taken out of their hands by a job loss.

So for many, the only way to delay the start of Social Security is to start tapping retirement savings. If that's the choice, is it worth doing?

[See: 13 Tips for Singles Nearing Retirement .]

It's complicated, says Paul Ruedi, CEO at Ruedi Wealth Management in Champaign, Illinois, noting that tax issues further cloud the picture. Experts say married couples must also weigh how a recipient's decision would affect the surviving spouse later.

"No rule of thumb is optimal," Ruedi says. "Anyone that says there is a rule of thumb just is not aware of the holistic nature of claiming (benefits)."

Ilene Davis, a planner with Financial Independence Services in Cocoa, Florida, says there is no single right answer. "Life expectancy and how long you think full benefits will be paid, to me, are key factors I consider when helping clients with this decision," she says.

How to Invest in Retirement

Delaying Social Security can indeed pay off, with the benefit jumping by up to 8 percent a year between age 62 and 70, depending on your age. If you were born in 1954 and were entitled to $2,000 at your "full retirement age" of 66, you'd get just $1,500 by starting when you become eligible this year at 62. Wait until 70 and you'd get $2,640.

Of course, you'll get more payments if you start earlier. In the example, the recipient could get $72,000 over the four years before turning 66. It would take 144 months, or 12 years, for the extra $500 from starting at 66 to offset that $72,000. So if you live beyond 78, delay would pay off; die younger and it would not.

But adding other retirement savings to the calculation makes things much trickier. It boils down to which "investment" pays off better: letting your Social Security benefit grow by delaying the start, or letting your investments grow by starting Social Security so you don't have to tap the retirement accounts.

The Social Security delay "earns" 8 percent a year, guaranteed, and it might be hard to do as well with your retirement savings without taking a lot of risk.

[See: 6 Strategies To Avoid Working in Retirement .]

For a simplified example, you could take $72,000 from your retirement accounts to postpone starting Social Security from 62 to 66. Or you could start Social Security at 62 and leave the $72,000 untouched, hoping it would earn enough later to offset the $500 a month given up with the early start. To produce $500 a month, the $72,000 would need a return of 6.9 percent. You might get a return that big in stocks, but probably would not find an interest-bearing account that generous, or as dependable as the gain from delaying Social Security. From this perspective, leaving that sum alone is not likely to offset the penalty for starting Social Security early, so it would pay to tap the retirement account in order to get the bigger Social Security benefit from delaying.

But there are other things to take into account. If one spouse in a married couple has a shorter life expectancy, he or she would probably be wise to start benefits earlier rather than later, says Jay Messing, senior director of wealth planning for Wells Fargo Private Bank in New York.

A surviving spouse in a married couple has the right to claim his or her benefit, or the benefit of the deceased spouse. In this case, Messing says, it would pay for the spouse likely to pass away first to maximize his or her benefit by delaying. Traditionally, this has been a husband who is older and earned more.

Still another factor, he says, is whether the beneficiary will work after starting benefits. This year, a person who has not reached full retirement age has the benefit trimmed by $1 for every $2 earned above $15,720. It would probably not make sense in this case to start benefits before full retirement age.

How to Create a Foolproof Withdrawal Plan for Retirement Assets

Among the unknowns in this decision are how the Social Security rules might change as the government struggles with the shortfall forecast for the trust fund.

Also, investors need to consider the tax consequences of drawing money from various types of accounts – taxable accounts subject to long- or short-term capital gains tax, 401(k)s and IRAs taxed as income, or tax-free Roth accounts.

Finally, drawing a lot of taxable income can trigger a tax on part of your Social Security benefit. A married couple filing a joint return, for instance, is not taxed if "combined income" is below $32,000, but is taxed on half of the benefit if combined income is between $32,000 and $44,000. Combined income is adjusted gross income, plus non-taxable income like interest from municipal bonds, plus 50 percent of the Social Security benefit. With a higher combined income, up to 85 percent of the Social Security benefit is taxed.

Generally, delaying the start of Social Security will make one wealthier in the long run. But so many factors must be considered it may pay to hire a professional planner to help figure it out.

[See: 10 Questions to Ask Before You Hire a Financial Advisor .]

"The best path," Ruedi says, "is to find an advisor who can manage all of these variables in concert to determine the optimal claiming strategy, in the backdrop of an overall retirement income strategy."

6 Strategies to Avoid Working in Retirement

Compare Offers

Compare Offers